Emotion estimation apparatus

ABSTRACT

An emotion estimation apparatus includes a recording section that records one or more events that cause a change in an emotion of a person and prediction information for predicting, for each event, an occurrence of the event; an event predicting section that predicts the occurrence of the event, based on detection of the prediction information; and a frequency setting section that sets a frequency with which an estimation of the emotion is performed. If the occurrence of the event is predicted by the event predicting section, the frequency setting section sets the frequency to be higher than in a case where the occurrence of the event is not predicted, and also sets the frequency based on the content of the event.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of the U.S. patent application Ser.No. 16/665,491, filed on Oct. 28, 2019, which is based upon and claimsthe benefit of priority from Japanese Patent Application No.2018-203563, filed on Oct. 30, 2018. The entire subject matter of thesepriority documents, including specification, claims and drawingsthereof, is incorporated by reference herein.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an emotion estimation apparatus thatestimates an emotion of a person.

Description of the Related Art

Emotion estimation apparatuses that estimate the emotion of a person arebeing developed in various fields. Japanese Laid-Open Patent PublicationNo. 2016-071577 discloses an apparatus that captures an image of theface of a driver with a camera and determines whether the driver is in acareless state based on the number of changes of his or her expression(changes of prescribed locations of the face) within a prescribed period(e.g. within three minutes).

SUMMARY OF THE INVENTION

If the frequency of the emotion estimation is high, the accuracy of thecomprehension of the emotion of the person can be made high, but thisalso results in a large computational load.

The present invention takes the above problem into consideration, and itis an object of the present invention to provide an emotion estimationapparatus that can suitably comprehend the emotion of a person whilealso restricting an increase in the computational load.

An aspect of the present invention is an emotion estimation apparatuscomprising a recording section configured to record predictioninformation for, for each event that causes a change in an emotion of aperson, predicting an occurrence of the event; an event predictingsection configured to predict the occurrence of the event, based ondetection of the prediction information; and a frequency setting sectionconfigured to set a frequency with which an estimation of the emotion isperformed, wherein if the occurrence of the event is predicted by theevent predicting section, the frequency setting section sets thefrequency to be higher than in a case where the occurrence of the eventis not predicted, and also sets the frequency based on content of theevent.

According to the present invention, it is possible to suitablycomprehend the emotion of a person while also restricting an increase inthe computational load.

The above and other objects, features, and advantages of the presentinvention will become more apparent from the following description whentaken in conjunction with the accompanying drawings, in which apreferred embodiment of the present invention is shown by way ofillustrative example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an emotion estimation apparatus accordingto a first embodiment;

FIG. 2 is a process flow performed by the first embodiment and a secondembodiment;

FIG. 3 is a block diagram of an emotion estimation apparatus accordingto the second embodiment;

FIG. 4 is a block diagram of an emotion estimation apparatus accordingto a third embodiment;

FIG. 5 is a process flow performed by the third embodiment;

FIG. 6 is a block diagram of an emotion estimation apparatus accordingto a fourth embodiment;

FIG. 7 is a process flow performed by the fourth embodiment and a fifthembodiment; and

FIG. 8 is a block diagram of an emotion estimation apparatus accordingto the fifth embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following describes detailed examples of preferred embodiments of anemotion estimation apparatus according to the present embodiment, whilereferencing the accompanying drawings.

In each embodiment described below, it is assumed that the emotionestimation apparatus is provided in a vehicle. In each embodiment, thetarget of the emotion estimation may be a specific person sitting at aprescribed position in the vehicle, or may be all people in the vehicle.In the following description, it is assumed that the emotion estimationis performed on the person sitting in the driver's seat (the driver).

The emotion estimation apparatus performs an emotion estimation of aperson with a set frequency. The emotion of a person changes after anevent occurs. Therefore, the emotion estimation apparatus predicts thepresence or lack of an occurrence of an event by detecting the presenceor lack of another event (referred to below as a preceding event) thathas a high possibility of occurring before the event occurs, and when itis predicted that the event will occur, sets the frequency to be higherthan in a case where it is predicted that the event will not occur.

1. First Embodiment

The following describes an emotion estimation apparatus 10 according toa first embodiment. In the first embodiment, a person talking to anotherperson in the vehicle (a conversation between a person and anotherperson) is imagined as the event that causes a change in the emotion ofthe person.

1.1. Configuration

The configuration of the emotion estimation apparatus 10 according tothe first embodiment is described using FIG. 1. The emotion estimationapparatus 10 includes an in-vehicle camera 12, a computing section 14,and a recording section 16.

The in-vehicle camera 12 may be provided at a position enabling imagecapturing of each seat, such as the dashboard or the ceiling in thevehicle, or the in-vehicle camera 12 may be provided for each seat. Thein-vehicle camera 12 acquires image information by capturing an image ofthe vehicle cabin (occupants), and outputs this image information to thecomputing section 14.

The computing section 14 includes a processor such as a CPU. Thecomputing section 14 realizes various functions by executing programsstored in the recording section 16. In the first embodiment, thecomputing section 14 functions as a recognizing section 20, an eventpredicting section 22, an identifying section 24, a frequency settingsection 26, and an emotion estimating section 28. The recognizingsection 20 recognizes a situation (position of an occupant, face of anoccupant, or the like) in the vehicle, based on the image informationacquired by the in-vehicle camera 12. The event predicting section 22predicts the occurrence of an event based on the detection of predictioninformation 32. The identifying section 24 identifies a person andanother person who is accompanying the person. The frequency settingsection 26 sets the frequency with which the emotion estimation isperformed. The emotion estimating section 28 performs the emotionestimation on the person based on the image information acquired by thein-vehicle camera 12. For example, the emotion estimating section 28performs the emotion estimation by judging subtle expressions. At thistime, the emotion estimating section 28 performs the emotion estimationwith the frequency set by the frequency setting section 26.

The recording section 16 includes a storage apparatus such as a ROM, aRAM, or the like. The recording section 16 records the programs to beexecuted by the computing section 14, various numerical values, and thefrequency of the emotion estimation (a prescribed frequency, a firstfrequency, and a second frequency), and also records event information30 and prediction information 32 in association with each other and aperson database 34 (also referred to below as a person DB 34). The eventinformation 30 is information defining an event. Furthermore, theprediction information 32 is information for predicting the occurrenceof an event indicated by the event information 30, i.e. informationdefining a preceding event. In the first embodiment, the eventinformation 30 is information defining the event of a conversationbetween the person and the other person, and the prediction information32 is information defining the preceding event of the face of the personor the line of sight of the person turning in the direction of the otherperson. The person DB 34 includes person information 36, other personinformation 38, and accompaniment history information 40. The personinformation 36 is biometric information of a person who is the target ofthe emotion estimation, and includes face image data, fingerprint data,and the like, for example. The other person information 38 isinformation linked to the person information 36, namely biometricinformation of other people who have a relationship with the personrecorded in the person information 36, and includes face image data,fingerprint data, and the like, for example. The accompaniment historyinformation 40 is data indicating, in time series, dates on which theperson and the other person were together.

1.2. Operation

The following describes the operation of the emotion estimationapparatus 10 according to the first embodiment using FIG. 2. The seriesof processes shown in FIG. 2 are performed periodically.

At step S1, the event predicting section 22 monitors the detectionsituation of the preceding event defined by the prediction information32, and predicts whether an event will occur. In this process, thein-vehicle camera 12 outputs the image information obtained by capturingan image of the inside of the vehicle to the computing section 14. Therecognizing section 20 recognizes the position of the person and theposition of the other person within the vehicle by an image recognitionprocess using the image information, and recognizes the orientation ofthe face of the person or the orientation of the line of sight of theperson through a face detection technique or line of sight detectiontechnique using the image information. The event predicting section 22identifies the direction of the other person with respect to the personand the orientation of the face of the person or the line of sight ofthe person, based on the recognition results of the recognizing section20. If the preceding event (turning of the face or the line of sight ofthe person in the direction of the other person) is detected, the eventpredicting section 22 predicts that the event (a conversation betweenthe person and the other person) will occur (step S1: YES). In thiscase, the process moves to step S3. On the other hand, if the precedingevent is not detected, the event predicting section 22 predicts that theevent will not occur (step S1: NO). In this case, the process moves tostep S2.

When the process moves from step S1 to step S2, the frequency settingsection 26 sets the prescribed frequency as the frequency for theemotion estimation. At this time, the computing section 14 outputsoperational instructions to the emotion estimation apparatus (not shownin the drawings) to perform the emotion estimation with the prescribedfrequency. After step S2, the series of processes is temporarilyfinished.

When the process moves from step S1 to step S3, the frequency settingsection 26 determines the relationship between the person and the otherperson, based on the identification results of the identifying section24. The identifying section 24 identifies the person bycross-referencing the image data of the face of the person recognized bythe recognizing section 20 with face image data of the personinformation 36 in the person DB 34. Furthermore, the identifying section24 identifies the other person by cross-referencing the image data ofthe face of the other person recognized by the recognizing section 20with face image data of the other person information 38 linked to theperson information 36 of the identified person. Yet further, thefrequency setting section 26 identifies the accompaniment historyinformation 40 linked to the other person information 38 and the personinformation 36 based on the identification result of the identifyingsection 24, and determines the relationship between the person and theother person. For example, the frequency setting section 26 calculatesthe number of times these people have accompanied each other, based onthe accompaniment history information 40. Then, if the number ofaccompaniments is greater than a count threshold, the frequency settingsection 26 determines that these people have a close relationship.Furthermore, if the number of accompaniments is less than or equal tothe count threshold, the frequency setting section 26 determines thatthese people do not have a close relationship. If the identifyingsection 24 cannot identify the other person, the frequency settingsection 26 determines that these people do not have a closerelationship. Instead, the frequency setting section 26 may determinethe relationship between these people according to the number ofaccompaniments within a prescribed time period. After step S3, theprocess moves to step S4.

At step S4, the frequency setting section 26 determines whether theother person is someone the person is highly considerate of. If thesepeople do not have a close relationship, the frequency setting section26 determines that the other person is someone the person needs to behighly considerate of (step S4: YES). In this case, the process moves tostep S5. On the other hand, if these people have a close relationship,the frequency setting section 26 determines that the other person is notsomeone that the person needs to be highly considerate of (step S4: NO).In this case, the process moves to step S6.

When the process moves from step S4 to step S5, the frequency settingsection 26 sets the frequency of the emotion estimation to be the firstfrequency, which is higher than the prescribed frequency. At this time,the computing section 14 outputs operational instructions to the emotionestimation apparatus (not shown in the drawings) to perform the emotionestimation with the first frequency. After step S5, the series ofprocesses is temporarily finished.

When the process moves from step S4 to step S6, the frequency settingsection 26 sets the frequency of the emotion estimation to be the secondfrequency, which is higher than the prescribed frequency and lower thanthe first frequency. At this time, the computing section 14 outputsoperational instructions to the emotion estimation apparatus (not shownin the drawings) to perform the emotion estimation with the secondfrequency. After step S6, the series of processes is temporarilyfinished.

As shown in step S4, step S5, and step S6, the frequency setting section26 sets the frequency based on the “event content” (whether theconversation is between people having a close relationship) of the eventthat is a conversation between the person and the other person.

The computing section 14 updates the accompaniment history information40 linked to the other person information 38 and the person information36. At this time, the computing section 14 newly registers, as theaccompaniment history information 40, a system date at the time when theperson and the other person are identified. However, in a case whereinformation for a date that is the same as the system date is alreadyrecorded in the accompaniment history information 40, the computingsection 14 does not update the accompaniment history information 40.

1.3. Modification of the First Embodiment

In the above description, it is imagined that the emotion estimationapparatus 10 is provided in a vehicle. Instead, the emotion estimationapparatus 10 according to the first embodiment may be provided in amoving body other than a vehicle. Furthermore, instead of a moving body,the emotion estimation apparatus 10 according to the first embodimentmay be provided in a personal computer (including a mobile PC and tabletPC) or a smartphone, for example.

2. Second Embodiment

The following describes the emotion estimation apparatus 10 according toa second embodiment. In the second embodiment, the person and the otherperson getting into the vehicle is imagined as an event that causes achange in the emotion of a person.

2.1. Configuration

The following describes the configuration of the emotion estimationapparatus 10 according to the second embodiment using FIG. 3. In thesecond embodiment, configurations that are the same as in the firstembodiment are given the same reference numerals, and descriptionsthereof are omitted. The emotion estimation apparatus 10 includes thein-vehicle camera 12, touch sensors 42, fingerprint sensors 44, thecomputing section 14, and the recording section 16.

The touch sensors 42 and the fingerprint sensors 44 are provided inrespective door handles. The touch sensor 42 detects contact with thedoor handles and outputs detection information to the computing section14. The fingerprint sensor 44 detects the fingerprint of the person orthe other person who has touched the door handle and outputs fingerprintinformation to the computing section 14.

In the second embodiment, the computing section 14 functions as therecognizing section 20, the event predicting section 22, the identifyingsection 24, and the frequency setting section 26.

In the second embodiment, the recording section 16 records the eventinformation 30 and the prediction information 32 in association witheach other, and also records the person DB 34. In the second embodiment,the event information 30 is information defining the event of the personand the other person getting into the vehicle, and the predictioninformation 32 is information defining the preceding event of the personand the other person touching the door handles.

2.2. Operation

In the same manner as in the first embodiment, the following describesthe operation of the emotion estimation apparatus 10 according to thesecond embodiment using FIG. 2. The series of processes shown in FIG. 2is performed periodically. In the following, the processes performed inthe second embodiment are described while focusing on steps (step S1 andstep S3) in which the processes different from those in the firstembodiment are performed.

At step S1, the event predicting section 22 monitors the detectionsituation of the preceding event defined by the prediction information32, and predicts whether the event will occur. In this process, theevent predicting section 22 identifies a contact state with the doorhandle, based on the detection results of the touch sensor 42 providedin the driver's side door and the touch sensor 42 provided in anotherdoor, e.g. the passenger side door. If the preceding event (the personand the other person touching the door handles) is detected, the eventpredicting section 22 predicts that the event (the person and the otherperson getting into the vehicle) will occur (step S1: YES). In thiscase, the process moves to step S3. On the other hand, if the precedingevent is not detected, the event predicting section 22 predicts that theevent will not occur (step S1: NO). In this case, the process moves tostep S2.

When the process moves from step S1 to step S3, the frequency settingsection 26 determines the relationship between the person and the otherperson, based on the identification result of the identifying section24. The identifying section 24 identifies the person bycross-referencing the fingerprint data detected by the fingerprintsensor 44 provided in the driver side door with the fingerprint data ofthe person information 36 in the person DB 34. Furthermore, theidentifying section 24 identifies the other person by cross-referencingthe fingerprint data detected by the fingerprint sensor 44 provided inthe passenger side door with the fingerprint data of the other personinformation 38 linked to the person information 36 of the identifiedperson. Yet further, the frequency setting section 26 determines therelationship between the person and the other person by identifying theaccompaniment history information 40 linked to the other personinformation 38 and the person information 36, based on theidentification results of the identifying section 24. The method fordetermining the relationship between the person and the other person canbe the same as in the first embodiment.

In the same manner as in the first embodiment, the frequency settingsection 26 sets the frequency based on the “event content” (getting intothe vehicle with someone the person has a close relationship with) ofthe event that is the person getting into the vehicle.

2.3. Modification of the Second Embodiment

The second embodiment may be combined with the first embodiment.

3. Third Embodiment

The following describes the emotion estimation apparatus 10 according toa third embodiment. In the third embodiment, a change in the drivingsituation is imagined as an event that causes a change in the emotion ofa person.

3.1. Configuration

The following describes the configuration of the emotion estimationapparatus 10 according to the third embodiment using FIG. 4. In thethird embodiment, configurations that are the same as in the firstembodiment are given the same reference numerals, and descriptionsthereof are omitted. The emotion estimation apparatus 10 includes thein-vehicle camera 12, a navigation apparatus 50, a velocity sensor 52,the computing section 14, and the recording section 16.

The navigation apparatus 50 includes a navigation computing section 54,a navigation recording section 56, a communicating section 58, apositioning section 60, and an HMI section 62. The navigation computingsection 54 includes a processor such as a CPU. The navigation recordingsection 56 includes a storage apparatus such as a ROM and RAM, andrecords map information. The communicating section 58 includes a firstreceiver that receives radio waves transmitted from a communicationterminal provided on the road, and a second receiver that receives radiowaves transmitted from a broadcast station. The positioning section 60includes a gyro sensor and a GPS receiver that receives radio wavestransmitted from a GPS satellite. The navigation apparatus 50 outputs,to the computing section 14, information such as traffic informationreceived by the respective receivers and the vehicle position determinedby the GPS receiver or the like. The velocity sensor 52 detects thetravel velocity of the vehicle and outputs this travel velocity to thecomputing section 14.

In the third embodiment, the computing section 14 functions as therecognizing section 20, the event predicting section 22, a riskestimating section 64, and the frequency setting section 26. The riskestimating section 64 estimates the amount of risk for the drivingsituation of the vehicle in which the person is riding.

In the third embodiment, the recording section 16 records the eventinformation 30 and the prediction information 32 in association witheach other, and also records a risk amount computation map 66. In thethird embodiment, the event information 30 is information defining theevent of a change in the driving situation, and the predictioninformation 32 is information defining the preceding event ofapproaching a left-right turning point. Here, “approaching” refers tothe distance between the vehicle position and the left-right turningpoint becoming less than or equal to a prescribed distance. The riskamount computation map 66 associates, in advance, a risk parameter (thenumber of traffic accidents, the number of traffic participants, thevehicle velocity, or the like) with the risk amount. Here, the “riskamount” refers to the amount of the travel risk at the left-rightturning point or following the left-right turning point. In thefollowing description, the risk amount is the amount of the travel riskat the left-right turning point.

3.2. Operation

The following describes the operation of the emotion estimationapparatus 10 according to the third embodiment using FIG. 5. The seriesof processes shown in FIG. 5 are performed periodically. Among theprocesses shown in FIG. 5, the processes of step S12, step S15, and stepS16 are the same as the processes of step S2, step S5, and step S6 shownin FIG. 2. The following describes the processes performed in the thirdembodiment while focusing on the processes of step S11, step S13, andstep S14, at which original processes are performed.

At step S11, the event predicting section 22 monitors the detectionsituation of the preceding event defined by the prediction information32, and predicts whether the event will occur. In this process, thenavigation apparatus 50 measures the newest vehicle position andmonitors the planned travel path set within a prescribed distance fromthe vehicle position. The prescribed distance is recorded in advance inthe navigation recording section 56. When a left-right turning pointpresent within a prescribed distance from the vehicle position isdetected, the navigation apparatus 50 outputs, to the computing section14, a detection signal indicating that a left-right turning point iswithin the prescribed distance. If the preceding event (approaching aleft-right turning point) is detected by the navigation apparatus 50,the event predicting section 22 predicts that the event (change in thedriving situation) will occur (step S11: YES). In this case, the processmoves to step S13. On the other hand, if the preceding event is notdetected by the navigation apparatus 50, the event predicting section 22predicts that the event will not occur (step S11: NO). In this case, theprocess moves to step S12.

When the process moves from step S11 to step S13, the frequency settingsection 26 identifies the risk amount based on the estimation result ofthe risk estimating section 64. In this process, the navigationapparatus 50 acquires, from the broadcast station or an externalcommunication terminal via the communicating section 58, trafficinformation, e.g. information concerning the number of trafficparticipants or the number of traffic accidents, for the road containingthe left-right turning point, and outputs this information to thecomputing section 14. Furthermore, the velocity sensor 52 outputs thedetected vehicle velocity to the computing section 14. The riskestimating section 64 estimates the risk amount corresponding to atleast one of the risk parameters (the number of traffic participants,the number of traffic accidents, and the vehicle velocity) using therisk amount computation map 66. The frequency setting section 26identifies the risk amount estimated by the risk estimating section 64as the risk amount of the road containing the left-right turning point.

At step S14, the frequency setting section 26 compares the risk amountto a reference value. Information concerning the reference value isrecorded in advance in the recording section 16. If the risk amount isgreater than the reference value (step S14: YES), the process moves tostep S15. On the other hand, if the risk amount is less than or equal tothe reference value (step S14: NO), the process moves to step S16.

As shown in step S14, step S15, and step S16, the frequency settingsection 26 sets the frequency based on the “event content” (whether thechange in the driving situation causes the risk amount to become higherthan the reference value) of the event that is a change in the drivingsituation.

3.3. Modifications of the Third Embodiment 3.3.1. First Modification

The third embodiment may be combined with at least one of the firstembodiment and the second embodiment.

3.3.2. Second Modification

The recording section 16 may record in advance a map in which the riskamount and the frequency are associated. The map associates higherfrequencies with higher risk amounts. In this case, the frequencysetting section 26 obtains the frequency corresponding to the riskamount from the map.

4. Fourth Embodiment

The following describes the emotion estimation apparatus 10 according toa fourth embodiment. In the fourth embodiment, notification of guidanceto a shop being provided to a person riding in the vehicle (notificationof information) is imagined as an event that causes a change in theemotion of a person.

4.1. Configuration

The following describes the configuration of the emotion estimationapparatus 10 according to the fourth embodiment using FIG. 6. In thefourth embodiment, configurations that are the same as in the firstembodiment are given the same reference numerals, and descriptionsthereof are omitted. The emotion estimation apparatus 10 includes thein-vehicle camera 12, the navigation apparatus 50, the computing section14, and the recording section 16.

The navigation recording section 56 of the navigation apparatus 50records registered shop information 68 and distance information 70 (afirst distance and a second distance). The registered shop information68 is information concerning a shop registered by the person, e.g.information of a preferred shop (a shop in line with the person'spreferences), and includes the name of the shop and positioninformation. The first distance in the distance information 70 isinformation defining the timing at which notification of guidance to theshop is provided. The second distance in the distance information 70 isinformation defining the timing at which a prediction of thisnotification is performed. The first distance is less than the seconddistance. In the present embodiment, if the distance between the vehicleposition measured by the positioning section 60 and the position of theshop registered as the registered shop information 68 is less than orequal to the second distance, the navigation apparatus 50 outputs adetection signal indicating that a preferred shop is nearby, and if theabove distance is less than or equal to the first distance, thenavigation apparatus 50 provides notification indicating that thevehicle is near the shop, via the HMI section 62.

In the fourth embodiment, the computing section 14 functions as theevent predicting section 22, the recognizing section 20, the identifyingsection 24, a confidence determining section 72, and the frequencysetting section 26. The confidence determining section 72 determines thedegree of confidence of the notification content provided by the HMIsection 62 of the navigation apparatus 50.

In the fourth embodiment, the recording section 16 records the eventinformation 30 and the prediction information 32 in association witheach other, and also records a person-shop database 74 (also referred tobelow as a person-shop DB 74) and a first confidence computation map 80.In the fourth embodiment, the event information 30 is informationdefining the event of information (shop guidance) notification, and theprediction information 32 is information defining the preceding event ofapproaching a preferred shop. The person-shop DB 74 includes the personinformation 36, shop information 76, and shop entrance historyinformation 78. The shop information 76 is information linked to theperson information 36, and includes information concerning the names ofshops, types of shops, positions of shops, and the like visited in thepast by the person registered in the person information 36. The shopentrance history information 78 is information linked to the shopinformation 76 and the person information 36, and is data indicating, intime series, the dates that the person visited the shops. The firstconfidence computation map 80 associates in advance a degree ofconfidence parameter (the number of entrances to the shop, the shopentrance frequency, or the like) and a degree of confidence. Here,“degree of confidence” is a measure of whether or not the notificationcontent is appropriate.

4.2. Operation

The following describes the operation of the emotion estimationapparatus 10 according to the fourth embodiment using FIG. 7. The seriesof processes shown in FIG. 7 are performed periodically. Among theprocesses shown in FIG. 7, the processes of step S22, step S25, and stepS26 are the same as the processes of step S2, step S5, and step S6 shownin FIG. 2. The following describes the processes performed in the fourthembodiment while focusing on the processes of step S21, step S23, andstep S24, at which original processes are performed.

At step S21, the event predicting section 22 monitors the detectionsituation of the preceding event defined by the prediction information32, and predicts whether the event will occur. In this process, thenavigation apparatus 50 measures the newest vehicle position andmonitors the relative position of the vehicle to the shop registered inthe registered shop information 68. If the distance between the vehicleand the shop is less than or equal to the second distance (which isgreater than the first distance), the navigation apparatus 50 outputs,to the computing section 14, a detection signal indicating that theregistered shop is within the second distance, i.e. that a preferredshop is nearby. If the preceding event (approaching a preferred shop) isdetected by the navigation apparatus 50, the event predicting section 22predicts that the event (notification of information) will occur (stepS21: YES). In this case, the process moves to step S23. On the otherhand, if the preceding event is not detected by the navigation apparatus50, the event predicting section 22 predicts that the event will notoccur (step S21: NO). In this case, the process moves to step S22.

When the process moves from step S21 to step S23, the frequency settingsection 26 identifies the degree of confidence based on thedetermination result of the confidence determining section 72. In thisprocess, the navigation apparatus 50 outputs, to the computing section14, the registered shop information 68 concerning the shop for which thedistance from the vehicle has become less than or equal to the seconddistance. Furthermore, the in-vehicle camera 12 outputs, to thecomputing section 14, the image information obtained by capturing animage of the inside of the vehicle. The recognizing section 20recognizes the face of the person in the vehicle through an imagerecognition process using the image information. The identifying section24 identifies the person by cross-referencing the image data of the faceof the person recognized by the recognizing section 20 with the faceimage data of the person information 36 in the person-shop DB 74. Theconfidence determining section 72 searches for shop information 76matching the registered shop information 68, in the shop information 76linked to the person information 36 of the identified person.Furthermore, the confidence determining section 72 calculates the numberof entrances to the shop or the shop entrance frequency within aprescribed time period, based on the shop entrance history information78 linked to the shop information 76 found in the search. The confidencedetermining section 72 determines the degree of confidence correspondingto at least one of the confidence parameters (the number of shopentrances and the shop entrance frequency) using the first confidencecomputation map 80. The frequency setting section 26 identifies thedegree of confidence determined by the confidence determining section 72as the degree of confidence of the notification information.

At step S24, the frequency setting section 26 compares the degree ofconfidence to a reference value. The information concerning thereference value is recorded in advance in the recording section 16. Ifthe degree of confidence is less than the reference value (step S24:YES), the process moves to step S25. On the other hand, if the degree ofconfidence is greater than or equal to the reference value (step S24:NO), the process moves to step S26.

As shown in step S24, step S25, and step S26, the frequency settingsection 26 sets the frequency based on the “event content” (whether thenotification by the system has a high degree of confidence) of the eventthat is notification of information.

If the person has actually entered the shop, the computing section 14updates the shop entrance history information 78 linked to the shopinformation 76 and the person information 36. In this case, thecomputing section 14 newly registers, as the shop entrance historyinformation 78, a system date at the time when the vehicle has parked ordeparted from near the shop. The computing section 14 detects theparking or departure of the vehicle according to a manipulation of anignition switch, starter switch, or the like.

4.3. Modifications of the Fourth Embodiment 4.3.1. First Modification

The fourth embodiment may be combined with at least one of the firstembodiment to the third embodiment.

Furthermore, the frequency setting section 26 may set a unique frequencyfor each of “conversation between the person and the other person”, “theperson and the other person getting into the vehicle”, “change in thedriving situation”, and “notification of information”, which aredifferent types of event content.

4.3.2. Second Modification

The recording section 16 may record in advance a map in which the degreeof confidence and the frequency are associated. The map associates lowerfrequencies with higher degrees of confidence. In this case, thefrequency setting section 26 obtains the frequency corresponding to thedegree of confidence from the map.

5. Fifth Embodiment

The following describes the emotion estimation apparatus 10 according toa fifth embodiment. In the fifth embodiment, notification of arecommendation for refueling (recommended item) to a person riding inthe vehicle (notification of information) is imagined as an event thatcauses a change in the emotion of a person. In the present embodiment,the vehicle is imagined to be a gasoline-powered vehicle ordiesel-powered vehicle that uses gasoline, diesel oil, or the like asfuel.

5.1. Configuration

The following describes the configuration of the emotion estimationapparatus 10 according to the fifth embodiment using FIG. 8. In thefifth embodiment, configurations that are the same as in the firstembodiment are given the same reference numerals, and descriptionsthereof are omitted. The emotion estimation apparatus 10 includes thein-vehicle camera 12, the navigation apparatus 50, a fuel sensor 86, thecomputing section 14, and the recording section 16.

The fuel sensor 86 is provided in a fuel tank of the vehicle. The fuelsensor 86 detects the remaining amount of fuel in the fuel tank, andoutputs this remaining amount to the navigation apparatus 50 and thecomputing section 14.

The navigation recording section 56 of the navigation apparatus 50records remaining amount information 84 (a first remaining amount and asecond remaining amount). The first remaining amount in the remainingamount information 84 is information defining a timing for providingnotification recommending refueling. The second remaining amount in theremaining amount information 84 is information defining a timing forpredicting this notification. The first remaining amount is less thanthe second remaining amount. In the present embodiment, if the remainingamount of fuel detected by the fuel sensor 86 is less than or equal tothe first remaining amount, the navigation apparatus 50 providesnotification to the person in the vehicle that refueling is necessary,via the HMI section 62.

In the fifth embodiment, the computing section 14 functions as therecognizing section 20, the event predicting section 22, the confidencedetermining section 72, and the frequency setting section 26.

In the fifth embodiment, the recording section 16 records the eventinformation 30 and the prediction information 32 in association witheach other, and also records a second confidence computation map 88. Inthe fifth embodiment, the event information 30 is information definingthe event of notification of information (refueling recommendation), andthe prediction information 32 is information defining the precedingevent of the remaining amount of fuel becoming less than or equal to thesecond remaining amount. The second confidence computation map 88associates in advance a degree of confidence parameter (travelabledistance−distance to destination) and a degree of confidence.

5.2. Operation

In the same manner as in the fourth embodiment, the following describesthe operation of the emotion estimation apparatus 10 according to thefifth embodiment using FIG. 7. The series of processes shown in FIG. 7are performed periodically. The following describes the processesperformed in the fifth embodiment while focusing on the steps (step S21and step S23) in which the processes different from those in the fourthembodiment are performed.

At step S21, the event predicting section 22 monitors the detectionsituation of the preceding event defined by the prediction information32, and predicts whether the event will occur. In this process, the fuelsensor 86 detects the remaining amount of fuel and outputs thisremaining amount to the navigation apparatus 50 and the computingsection 14. If the remaining amount of fuel is less than or equal to thesecond remaining amount (which is greater than the first remainingamount), the navigation apparatus 50 outputs, to the computing section14, a signal indicating that notification of a recommendation to refuelis to be provided. If the preceding event (the remaining amount of fuelbecoming less than or equal to the second remaining amount) is detectedby the navigation apparatus 50, the event predicting section 22 predictsthat the event (notification of information) will occur (step S21: YES).In this case, the process moves to step S23. On the other hand, if thepreceding event is not detected by the navigation apparatus 50, theevent predicting section 22 predicts that the event will not occur (stepS21: NO). In this case, the process moves to step S22.

When the process moves from step S21 to step S23, the frequency settingsection 26 identifies the degree of confidence based on thedetermination result of the confidence determining section 72. In thisprocess, the confidence determining section 72 calculates the longestdistance that can be traveled without refueling, i.e. the travelabledistance, based on the remaining amount of fuel detected by the fuelsensor 86 and the fuel consumption recorded in the recording section 16.Furthermore, the confidence determining section 72 acquires the distancefrom the vehicle position measured by the navigation apparatus 50 to thedestination, and calculates the “travelable distance−distance to thedestination”. The confidence determining section 72 determines thedegree of confidence corresponding to the “travelable distance−distanceto the destination” using the second confidence computation map 88. Thefrequency setting section 26 identifies the degree of confidencedetermined by the confidence determining section 72 as the degree ofconfidence of the notification information.

In the same manner as in the fourth embodiment, the frequency settingsection 26 sets the frequency based on the “event content” (whether thenotification by the system has high degree of confidence) of the eventthat is the notification of information.

5.3. Modifications of the Fifth Embodiment 5.3.1. First Modification

The fifth embodiment may be combined with at least one of the firstembodiment to the fourth embodiment.

5.3.2. Second Modification

The recording section 16 may record in advance a map in which the degreeof confidence and the frequency are associated, and the frequencysetting section 26 may obtain the frequency corresponding to the degreeof confidence from the map. In this case, the map associates lowerfrequencies with higher degrees of confidence.

5.3.3. Third Modification

In the fifth embodiment, when the notification of information (refuelingrecommendation) is made, this notification may be accompanied by aquestion. For example, when recommending refueling to the person, thenavigation apparatus 50 may ask (first question) whether to set arefueling location. In response to this question, the person answers yesor no using the HMI section 62. Furthermore, when recommending refuelingto the person, the navigation apparatus 50 may ask (second question)about a desired refueling location. In response to this question, theperson sets a refueling location using the HMI section 62.

The frequency setting section 26 sets the first frequency in the case ofa question, such as the second question, which cannot be answered withyes or no, and sets the second frequency (which is less than the firstfrequency) in the case of a question, such as the first question, whichcan be answered with yes or no.

In the third modification, the frequency setting section 26 sets thefrequency based on the “event content” (whether the notification of theinformation can be answered with yes or no) of the event of notificationof information.

6. Other Embodiments

The frequency with which the emotion estimation is performed may bechanged at other timings. For example, the preceding event of a phonecall or mail reaching a mobile terminal of the person may be set as thepreceding event for the event of the person talking on his or her mobiletelephone or reading a message that reached the mobile terminal of theperson. In this case, the frequency of the emotion estimation is set tobe higher than the normal frequency in response to a phone call or mailreaching the mobile terminal of the person.

7. Inventions that can be Obtained from the Embodiments

The following is a record of the inventions that can be understood fromthe embodiments and modifications described above.

The present invention comprises: the recording section 16 configured torecord the prediction information 32 for, for each event that causes achange in the emotion of the person, predicting the occurrence of anevent; the event predicting section 22 configured to predict theoccurrence of the event, based on detection of the predictioninformation 32; and the frequency setting section 26 configured to setthe frequency with which the estimation of the emotion is performed,wherein if the occurrence of the event is predicted by the eventpredicting section 22, the frequency setting section 26 sets thefrequency to be higher than in a case where the occurrence of the eventis not predicted, and also sets the frequency based on the content ofthe event.

According to the above configuration, when the occurrence of an eventcausing a change in the emotion of the person is predicted, thefrequency is set to be higher than in a case where the occurrence of theevent is not predicted, and therefore it is possible to appropriatelycomprehend the emotion of the person. Furthermore, since the frequencyis set based on the event content, it is possible to stop the emotionestimation from being performed with a needlessly high frequency, and totherefore restrict an increase in the computational load of theapparatus. Yet further, since the emotion estimation is performed withthe normal frequency when the occurrence of the event is not predicted,the computational load of the emotion estimation apparatus 10 isrestricted from becoming large.

The present invention (first embodiment) may comprise: the identifyingsection 24 configured to identify the other person accompanying theperson, wherein the recording section 16 may record the predictioninformation 32 for predicting the occurrence of a conversation betweenthe person and the other person, and the frequency setting section 26may set the frequency based on a result of identification of the otherperson by the identifying section 24.

According to the above configuration, it is possible to predict theevent that causes a change in the emotion of the person, which is theconversation between the person and the other person. At this time,since the frequency is set based on the relationship between the personand the other person, it is possible to perform the emotion estimationwith a suitable frequency.

The present invention (second embodiment) may comprise: the identifyingsection 24 configured to identify the other person accompanying theperson, wherein the recording section 16 may record the predictioninformation 32 for predicting whether the person will get into thevehicle, and the frequency setting section 26 may set the frequencybased on a result of identification of the other person by theidentifying section 24.

According to the above configuration, it is possible to predict theevent that causes a change in the emotion of the person, which is theperson and the other person getting into the vehicle. At this time,since the frequency is set based on the relationship between the personand the other person, it is possible to perform the emotion estimationwith a suitable frequency.

In the present invention (first and second embodiments), the recordingsection 16 may record, for each of a plurality of the other persons, aresult of determination as to whether the person is highly considerateof the other person, and the frequency setting section 26 may set thefirst frequency if the person is highly considerate of the other person(step S5 of FIG. 2), and may set the second frequency, which is lowerthan the first frequency, if the person is not highly considerate of theother person (step S6 of FIG. 2).

The present invention (third embodiment) may comprise: the riskestimating section 64 configured to estimate the risk amount for thedriving situation of the vehicle in which the person is riding, whereinthe recording section 16 may record the prediction information 32 forpredicting a change in the driving situation of the vehicle, and thefrequency setting section 26 may set the frequency based on the riskamount at a change location or after the change in the driving situationof the vehicle estimated by the risk estimating section 64.

According to the above configuration, it is possible to predict theevent that causes a change in the emotion of the person, which is achange in the driving situation. At this time, since the frequency isset based on the risk amount, it is possible to perform the emotionestimation with a suitable frequency.

In the present invention (third embodiment), the frequency settingsection 26 may set the frequency to be higher as the risk amount isgreater.

For example, the frequency setting section 26 may set the firstfrequency if the risk amount is higher than a prescribed reference value(step S15 of FIG. 5), and may set the second frequency, which is lowerthan the first frequency, if the risk amount is less than or equal tothe reference value (step S16 of FIG. 5).

In the present invention (third embodiment), the risk estimating section64 may estimate the risk amount based on at least one of the number oftraffic accidents, the vehicle velocity, and the number of trafficparticipants.

The present invention (fourth and fifth embodiments) may comprise: anotifying section (HMI section 62) configured to provide notification ofprescribed information to the person; and the confidence determiningsection 72 configured to determine the degree of confidence of thecontent of the notification provided by the notifying section, whereinthe recording section 16 may record the prediction information 32 forpredicting provision of notification of the prescribed information, andthe frequency setting section 26 may set the frequency based on thedegree of confidence determined by the confidence determining section72.

According to the above configuration, it is possible to predict theevent that causes a change in the emotion of the person, which isnotifying the person about information. At this time, since thefrequency is set based on the degree of confidence, it is possible toperform the emotion estimation with a suitable frequency.

In the present invention (fourth and fifth embodiments), the frequencysetting section 26 may set the frequency to be higher as the degree ofconfidence is lower.

For example, the frequency setting section 26 may set the firstfrequency if the degree of confidence is lower than a prescribedreference value (step S25 of FIG. 7), and may set the second frequency,which is lower than the first frequency, if the degree of confidence isgreater than the reference value (step S26 of FIG. 7).

In the present invention (fourth embodiment), the confidence determiningsection 72 may determine the degree of confidence according to whetherthe content of the notification is in line with a preference of theperson.

In the present invention (second modification of the fifth embodiment),the content of the notification to the person may accompany a questionto the person, and the frequency setting section 26 may set the firstfrequency if the question is not a yes or no question, and set thesecond frequency, which is lower than the first frequency, if thequestion is a yes or no question.

The emotion estimation apparatus of the present invention is not limitedto the above described embodiments, and various configurations can beadopted without deviating from the scope of the present invention.

What is claimed is:
 1. An emotion estimation apparatus comprising: arecording section configured to record prediction information forpredicting an occurrence of notification of prescribed information; anevent predicting section configured to predict the occurrence of thenotification of the prescribed information, based on detection of theprediction information; a frequency setting section configured to set afrequency with which an estimation of an emotion of a person isperformed; a notifying section configured to provide the notification ofthe prescribed information to the person; and a confidence determiningsection configured to determine a degree of confidence of content of thenotification provided by the notifying section, wherein the frequencysetting section sets the frequency to be higher in a case where theoccurrence of the notification of the prescribed information ispredicted by the event predicting section than in a case where theoccurrence of the notification of the prescribed information is notpredicted by the event predicting section, and the frequency settingsection sets the frequency based on the degree of confidence determinedby the confidence determining section.
 2. The emotion estimationapparatus according to claim 1, wherein the frequency setting sectionsets the frequency to be higher as the degree of confidence is lower. 3.The emotion estimation apparatus according to claim 2, wherein theconfidence determining section determines the degree of confidenceaccording to whether the content of the notification is in line with apreference of the person.
 4. The emotion estimation apparatus accordingto claim 2, wherein the content of the notification to the personaccompanies a question to the person, and the frequency setting sectionsets a first frequency if the question is not a yes or no question, andsets a second frequency, which is lower than the first frequency, if thequestion is a yes or no question.